Papers by Samuel R. Bowman
New Protocols and Negative Results for Textual Entailment Data Collection (2020.emnlp-main)
Copied to clipboard
| Challenge: | Natural language inference data has proven useful in benchmarking and as pretraining data for tasks requiring language understanding. |
| Approach: | They propose four alternative protocols to improve annotation quality and diversity . they use 8.5k-example training sets to compare different protocols . |
| Outcome: | The proposed protocols improve the ease of training and quality of the examples. |
Does Putting a Linguist in the Loop Improve NLU Data Collection? (2021.findings-emnlp)
Copied to clipboard
Alicia Parrish, William Huang, Omar Agha, Soo-Hwan Lee, Nikita Nangia, Alexia Warstadt, Karmanya Aggarwal, Emily Allaway, Tal Linzen, Samuel R. Bowman
| Challenge: | Many datasets for training and evaluating natural language understanding (NLU) models contain systematic artifacts that are identified only after data collection is complete. |
| Approach: | They propose to have linguists identify artifacts and gaps in the data and communicate with non-expert crowdworkers to adjust task instructions and incentives. |
| Outcome: | The proposed protocol does not increase accuracy on out-of-domain test sets, and adds a chatroom does not. |
Precise Task Formalization Matters in Winograd Schema Evaluations (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent results report a surge in performance to nearhuman levels on the Winograd Schema Challenge (WSC) however, variations in task formulation across papers and evaluations makes it hard to understand the true degree of recent progress. |
| Approach: | They propose to use a model with multiple choice to frame the task as multiple choice and reuse a pretrained language modeling head to mitigate the model's extreme sensitivity to hyperparameters. |
| Outcome: | The proposed frameworks improve the model's reasoning ability by framing the task as multiple choice and reuse of a pretrained language modeling head. |
Comparing Test Sets with Item Response Theory (2021.acl-long)
Copied to clipboard
Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, Samuel R. Bowman
| Challenge: | Recent results from large pretrained models show that many datasets are saturated and unlikely to detect further progress. |
| Approach: | They evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. |
| Outcome: | The proposed datasets are saturated and unlikely to detect future improvements. |
Crowdsourcing Beyond Annotation: Case Studies in Benchmark Data Collection (2021.emnlp-tutorials)
Copied to clipboard
| Challenge: | Developing a theory of crowdsourcing for practical language problems remains an open challenge . |
| Approach: | This tutorial exposes NLP researchers to data collection crowdsourcing methods and principles through case studies. |
| Outcome: | This tutorial exposes NLP researchers to various data collection crowdsourcing methods and practices through case studies. |
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021.naacl-main)
Copied to clipboard
| Challenge: | Evaluation for many natural language understanding (NLU) tasks is broken due to unreliable and biased systems scoring so high on standard benchmarks. |
| Approach: | They argue that current benchmarks fail at four criteria for evaluation . they argue that adversarial data collection does not address the causes of failures . |
| Outcome: | The proposed frameworks fail at four criteria, and adversarial data collection does not address the causes of these failures, the authors argue . restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, reliability with which they are annotated, their size, and the ways they handle social bias. |
BLiMP: The Benchmark of Linguistic Minimal Pairs for English (2020.tacl-1)
Copied to clipboard
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, Samuel R. Bowman
| Challenge: | Recent studies have examined how linguistic knowledge of language models (LMs) varies across English phenomena. |
| Approach: | They propose a benchmark to evaluate linguistic knowledge of language models on major grammatical phenomena in English. |
| Outcome: | The proposed benchmark evaluates the linguistic knowledge of language models on major grammatical phenomena in English. |
Identifying and Reducing Gender Bias in Word-Level Language Models (N19-3)
Copied to clipboard
| Challenge: | Existing discriminatory biases in training data can be amplified by models . text corpora exhibit socially problematic biase . |
| Approach: | They propose a metric to measure gender bias and a regularization loss term to minimize embeddings onto an embeddable subspace that encodes gender. |
| Outcome: | The proposed method reduces gender bias up to an optimal weight assigned to the loss term, and the model becomes unstable as the perplexity increases. |
On Measuring Social Biases in Sentence Encoders (N19-1)
Copied to clipboard
| Challenge: | Word embeddings such as word2vec and GloVe exhibit human-like implicit biases based on gender, race, and other social constructs. |
| Approach: | They propose a simple generaliza test to measure bias in word embeddings by comparing two sets of target-concept words to two sets . |
| Outcome: | The proposed test shows that word2vec and word2Ve exhibit human-like implicit biases based on gender, race, and other social constructs. |
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? (2020.acl-main)
Copied to clipboard
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, Samuel R. Bowman
| Challenge: | Unsupervised pretraining has recently pushed the state of the art on many natural language understanding tasks. |
| Approach: | They perform a large-scale survey on a pretrained RoBERTa model with 110 intermediate-target task combinations and 25 probing tasks to reveal the specific skills that drive transfer. |
| Outcome: | The proposed model is trained on 110 intermediate-target task combinations and compared with 25 probing tasks to reveal the specific skills that drive transfer. |
What Do NLP Researchers Believe? Results of the NLP Community Metasurvey (2023.acl-long)
Copied to clipboard
Julian Michael, Ari Holtzman, Alicia Parrish, Aaron Mueller, Alex Wang, Angelica Chen, Divyam Madaan, Nikita Nangia, Richard Yuanzhe Pang, Jason Phang, Samuel R. Bowman
| Challenge: | Getting sociological beliefs wrong can slow research and lead to wasted effort, missed opportunities, and needless fights. |
| Approach: | They present the results of the NLP Community Metasurvey, run from May to June 2022. |
| Outcome: | The NLP community metasurvey elicited opinions on controversial issues from May to June 2022. |
SocioProbe: What, When, and Where Language Models Learn about Sociodemographics (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have outperformed other models on a wide range of tasks . however, there is still little understanding of their knowledge of higher-level aspects of language . |
| Approach: | They investigate whether pre-trained language models have knowledge of sociodemographics . they use traditional probing techniques to probe the knowledge of single-GPU PLMs based on multiple English data sets . |
| Outcome: | The results show that pre-trained language models outperform other models on a wide range of tasks. |
Investigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs (D19-1)
Copied to clipboard
Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, Samuel R. Bowman
| Challenge: | Recent work evaluating sentence representation models' knowledge of grammar has been slower to emerge. |
| Approach: | They propose five experimental methods inspired by prior work evaluating pretrained sentence representation models to examine their grammatical knowledge. |
| Outcome: | The proposed methods show that the model has significant knowledge of the licensing environment but its success varies widely across different methods. |
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? (2021.acl-long)
Copied to clipboard
| Challenge: | Despite the importance of datasets for natural language understanding, there has been little attention on crowdsourcing methods for collecting datasets. |
| Approach: | They compare the effectiveness of crowdsourcing methods for boosting NLU example difficulty with training crowdworkers instead of expert judgments. |
| Outcome: | The proposed method is ineffective for boosting NLU example difficulty, but it is not effective for training crowdworkers and qualifying workers based on expert judgments. |
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing summarization datasets often have issues that seriously limit their usability. |
| Approach: | They propose a faster but more straightforward approach to developing summarization benchmark data . they use a protocol that hires highly-qualified contractors to read stories and write original summaries from scratch . |
| Outcome: | The proposed protocol is faster but more straightforward than scraping summaries from everyday text. |
jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models (2020.acl-demos)
Copied to clipboard
Yada Pruksachatkun, Phil Yeres, Haokun Liu, Jason Phang, Phu Mon Htut, Alex Wang, Ian Tenney, Samuel R. Bowman
| Challenge: | jiant is an open source toolkit for conducting multitask and transfer learning experiments on English NLU tasks. |
| Approach: | They introduce jiant, an open source toolkit for conducting multitask and transfer learning experiments on English NLU tasks. |
| Outcome: | The proposed toolkit reproduces published performance on GLUE and SuperGLUE tasks. |
Neural Unsupervised Parsing Beyond English (D19-61)
Copied to clipboard
| Challenge: | Unsupervised parsing is a task that can be learned without substantial prior knowledge. |
| Approach: | They train an unsupervised model for Arabic, Chinese, English, and German to learn syntactic structure from unlabeled text. |
| Outcome: | The PRPN architecture outperforms trivial baselines and acquires at least some parsing ability for all languages. |
Instruction Induction: From Few Examples to Natural Language Task Descriptions (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models can perform unseen tasks by conditioning on a few input-output demonstrations, but task inference is implicit and the ability of models to explicitly reason about it remains unexplored. |
| Approach: | They propose an instruction induction challenge in which a model is asked to generate a natural language instruction that fits a set of labeled examples. |
| Outcome: | The proposed model achieves 65.7% of human performance while the original model only reaches 9.8% of human performances. |
CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models use cultural biases implicitly, causing harm . identifying and quantifying learnt biase enables us to measure progress . |
| Approach: | They propose a benchmark to measure social bias in pretrained language models . they use 1508 examples that cover stereotypes dealing with nine types of bias . |
| Outcome: | The proposed benchmark focuses on stereotypes about historically disadvantaged groups and contrasts them with advantaged groups. |
(QA)2: Question Answering with Questionable Assumptions (2023.acl-long)
Copied to clipboard
| Challenge: | a number of questions contain questionable assumptions, such as when did Marie Curie discover Uranium, that cannot be answered as a true when question. |
| Approach: | They propose an open-domain evaluation dataset that can detect questionable assumptions . they propose a method that can be used to produce adequate responses for questions with questionable assumption. |
| Outcome: | The proposed model detects questionable assumptions and produces adequate responses for both types of questions. |
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)
Copied to clipboard
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman
| Challenge: | State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text. |
| Approach: | They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling. |
| Outcome: | The proposed model can be used to train sentences on language modeling tasks. |
Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set (D19-1)
Copied to clipboard
| Challenge: | Using development sets for low-resource training is often more effective . however, some studies show that early stopping can overestimate performance . |
| Approach: | They find that early stopping on a development set is more effective than using all available data for training. |
| Outcome: | The proposed model overestimates accuracy over languages and tasks by 1.4% compared to a more realistic set of training epochs. |
Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark (P19-1)
Copied to clipboard
| Challenge: | GLUE is a suite of language understanding tasks that has seen dramatic progress in the past year . average performance on the benchmark is 83.9, state of the art at the time of writing . |
| Approach: | They use crowdsourcing to measure human performance on a set of language understanding tasks and 20 examples to determine whether there is room for improvement. |
| Outcome: | The GLUE benchmark outperforms state-of-the-art models on six of the nine tasks and achieves an average score of 87.1. |
Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options (2020.aacl-main)
Copied to clipboard
| Challenge: | Large-scale natural language inference datasets are available for non-expert crowdsourcing. |
| Approach: | They propose to use crowdworker writing to create candidate pairs for annotation . they find crowdworkers writing is the best known protocol for entailment data . |
| Outcome: | The proposed protocol outperforms the baseline protocol in evaluations of generalization and transfer learning. |
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually) (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pretraining on self-supervised linguistic tasks is effective for learning features helpful for language understanding, but it requires more data to learn to prefer linguistic generalizations over surface ones. |
| Approach: | They propose a set of 20 ambiguous binary classification tasks to test whether a pretrained model prefers linguistic or surface generalizations. |
| Outcome: | The proposed model can learn to represent linguistic features with little pretraining data, but requires far more data to learn to prefer linguistic generalizations over surface ones. |
English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too (2020.aacl-main)
Copied to clipboard
Jason Phang, Iacer Calixto, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, Samuel R. Bowman
| Challenge: | a study of intermediate-task training in monolingual English shows that it improves model performance on non-English language understanding tasks. |
| Approach: | They evaluate whether English intermediate-task training is still helpful on non-English target tasks . BUCC and Tatoeba sentence retrieval tasks see large improvements . |
| Outcome: | The proposed model outperforms existing models on non-English language understanding tasks. |
When Do You Need Billions of Words of Pretraining Data? (2021.acl-long)
Copied to clipboard
| Challenge: | Pretrained language models (LMs) are dominated by models that can encode billions of words. |
| Approach: | They use classifier probing, information-theoretic probing and unsupervised relative acceptability judgments to evaluate model ability. |
| Outcome: | The proposed models require only about 10M to 100M words to learn to encode most syntactic and semantic features. |
ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning (2023.acl-short)
Copied to clipboard
| Challenge: | Negation is a ubiquitous but complex linguistic phenomenon that poses a significant challenge for NLP systems. |
| Approach: | They propose a benchmark that measures how well models handle natural language negation . they extend ScoNe-NLI to embed negation reasoning in short narratives . |
| Outcome: | The proposed model can reason about negation, but struggles to do so on NLI examples outside of its core pretraining regime. |
Discovering Language Model Behaviors with Model-Written Evaluations (2023.findings-acl)
Copied to clipboard
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan
| Challenge: | Prior work creates evaluations with crowdwork or existing data sources, which are not always available. |
| Approach: | They generate evaluations automatically with language models (LMs) using crowdwork or existing data sources to find out how they behave . |
| Outcome: | The results show that large LMs repeat back a dialog user’s preferred answer and express greater desire to pursue concerning goals like resource acquisition and goal preservation. |